Upload Babylon Native visualization-test images as pipeline artifacts#18456
Upload Babylon Native visualization-test images as pipeline artifacts#18456bghgary wants to merge 4 commits into
Conversation
The NativeTests job runs BabylonNative's validation_native.js against
the latest Babylon.js JS, but never uploaded the rendered PNGs (Results/)
or red-overlay diffs (Errors/) that validation_native.js writes next to
Playground.exe. Failure log was the only visible signal -- impossible to
tell whether the regression is a real visual change vs a one-pixel
threshold blip without re-running locally against a matching native
binary.
Mirror BabylonNative's own win32 workflow (actions/upload-artifact@v6
twice in .github/workflows/build-win32.yml) using two
PublishPipelineArtifact@1 steps:
- native-rendered-pictures -- Results/, uploaded on not(canceled()),
matches saveResult = true behavior in validation_native.js (every
test always writes its render so we get a green-run baseline too).
- native-error-pictures -- Errors/, uploaded only on failed(), matches
when validation_native.js actually writes diff overlays.
PublishPipelineArtifact@1 has no "ignore if missing" flag, so a one-off
New-Item -Force step ensures both dirs exist before publish (Errors/
won't exist on a fully-green run).
There was a problem hiding this comment.
Pull request overview
This PR updates the NativeTests CI job to preserve BabylonNative visual test output as Azure DevOps pipeline artifacts, making native rendering regressions easier to inspect after CI failures.
Changes:
- Ensures the native
Results/andErrors/output directories exist before artifact publishing. - Publishes rendered native visualization outputs on non-canceled runs.
- Publishes native error/diff images only when the job has failed.
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18456/merge/index.html#WGZLGJ#4600 Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves): https://playground.babylonjs.com/?snapshot=refs/pull/18456/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/18456/merge#BCU1XR#0 If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools. |
|
WebGL2 visualization test reporter: |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
Visualization tests for WebGPU |
bghgary
changed the title
…yground.exe The PublishPipelineArtifact@1 tasks were pointing at ..\BabylonNativeNightlyPlayground\RelWithDebInfo\Results and ...\Errors, but TestUtils.getOutputDirectory() on Win32 returns the parent of the module directory, so validation_native.js actually writes to ..\BabylonNativeNightlyPlayground\Results\ (not under RelWithDebInfo). First PR-build (53411) confirmed: the visualization-tests step failed (EXT_mesh_gpu_instancing, 205370 pixels diff), Upload Rendered Pictures ran and reported 'Processed 0 files from ...\\RelWithDebInfo\\Results', producing an empty artifact. Move the targetPath one level up to match the actual output location and we'll get the PNGs.
|
WebGL2 visualization test reporter: |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
Visualization tests for WebGPU |
|
Visualization tests for WebGPU |
|
WebGL2 visualization test reporter: |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
|
WebGL2 visualization test reporter: |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
Visualization tests for WebGPU |
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
[Created by Copilot on behalf of @bghgary]
Context
The
NativeTestsjob inci-monorepo.ymldownloads BabylonNative nightly test artifacts (pipeline 36), replaces their bundledbabylon*.jswith the freshly built snapshot, and runsPlayground.exe app:///Scripts/validation_native.jsplusUnitTests.exe.When pixel-diff regressions occur (e.g. the recent OpenPBR/glTF surface-tinting regression in #18430),
validation_native.jswrites the rendered PNG toResults/<reference>.pngand a red-overlay diff toErrors/<reference>.pngnext toPlayground.exe-- but nothing in the pipeline uploaded those directories. The failure log was the only visible signal, making it impossible to tell whether a regression is a real visual change vs. a threshold blip without re-running the test locally against a matching native binary.Change
Mirror the upload pattern from BabylonNative's own win32 workflow (
.github/workflows/build-win32.yml,actions/upload-artifact@v6twice) using the ADO equivalentPublishPipelineArtifact@1:native-rendered-pictures--Results/, uploaded onnot(canceled()). MatchessaveResult = trueinvalidation_native.js: every test always writes its render, so green runs upload too and give a baseline.native-error-pictures--Errors/, uploaded only onfailed(). Matches whenvalidation_native.jsactually writes diff overlays.PublishPipelineArtifact@1has noif-no-files-found: ignoreflag, so a one-offNew-Item -Forcestep ensures both directories exist before publish (Errors/legitimately won't exist on a fully-green run).Verification
Can be verified end-to-end only after merge: the next native-test failure runs through this job and surfaces the diff PNGs as a downloadable artifact on the build. Until then, the green-run path is exercised by every passing build of this PR itself --
native-rendered-picturesshould appear on the ADO run for this PR's CI.